{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "jekyll-raw"
    ]
   },
   "source": [
    "# Automatically building, previewing, and pushing your book with CircleCI\n",
    "\n",
    "CircleCI is a continuous integration service that lets you run various commands\n",
    "every time a new change is made to a repository. This can be used to **build your book**,\n",
    "**preview changes**, and even **push live HTML** as you update your book content.\n",
    "\n",
    "In order to accomplish each of these, we'll use a CircleCI configuration file. This\n",
    "is a YAML file that is used to tell Circle what to do with your repository.\n",
    "\n",
    "In each case, the expectation is that your **master branch holds your book content**.\n",
    "\n",
    "We'll step through each piece of a sample CircleCI configuration to show you how\n",
    "to accomplish this.\n",
    "\n",
    "## Preparing CircleCI\n",
    "\n",
    "First of all, you should set up your CircleCI account to start running CI jobs for\n",
    "your book repository. Follow these steps:\n",
    "\n",
    "* **Tell CircleCI to build your repository**. To do so,\n",
    "  [follow the CircleCI github integration instructions](https://circleci.com/integrations/github/).\n",
    "* **Tell CircleCI to build pull-requests to your repository**. To do so,\n",
    "  go to\n",
    "  \n",
    "  ```\n",
    "  https://circleci.com/gh/{{YOUR-GITHUB-ORG}}/{{YOUR-GITHUB-REPO}}/edit#advanced-settings\n",
    "  ```\n",
    "  \n",
    "  Find the \"*build forked pull requests*\" section, and switch it to **ON**.\n",
    "\n",
    "Now, CircleCI will start watching your repository. If it finds a Circle configuration\n",
    "file (more information on this below), it'll run a CI job according to the configuration\n",
    "it finds.\n",
    "\n",
    "You can copy/paste the empty CircleCI configuration here. This won't actually do anything\n",
    "but we'll add to it later:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "full_width"
    ]
   },
   "source": [
    "```yaml\n",
    "# Tell CircleCI which version of its API you wish to use\n",
    "version: 2.1\n",
    "jobs:\n",
    "    # Jobs define the different parts of your CircleCI workflow\n",
    "    # They can depend on one another, use pieces from one another, etc.\n",
    "    # We'll fill them in later\n",
    "workflows:\n",
    "    # Workflows tell CircleCI the order in which to run your jobs\n",
    "commands:\n",
    "    # Commands are re-usable chunks of steps that can be shared across jobs\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start filling out this template with a few jobs. In each case, we'll\n",
    "use a Python Docker image to both build each page's HTML, and build the book's\n",
    "HTML."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1: Build each page's HTML\n",
    "\n",
    "First you'll build each page's HTML. This is the initial conversion from\n",
    "`ipynb`, `md`, etc files. We'll use a Python container for this in order to\n",
    "use the Jupyter Book command-line interface with `jupyter-book build`.\n",
    "\n",
    "You can build your book's HTML files and preview them using CircleCI artifacts.\n",
    "To do this, you'll need to use two CircleCI jobs:\n",
    "   \n",
    "We'll need to persist the results of this step so that they are available in\n",
    "subsequent steps. Here's the CircleCI configuration that will accomplish this, which you can\n",
    "add to the skeleton configuration you've created above:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "full_width"
    ]
   },
   "source": [
    "```yaml\n",
    "jobs:\n",
    "  build_page_files:\n",
    "    docker:\n",
    "      - image: circleci/python:3.6-stretch\n",
    "    steps:\n",
    "      # Get our data and merge with upstream\n",
    "      - checkout\n",
    "\n",
    "      # Install the packages needed to build our documentation\n",
    "      # This will depend on your particular package!\n",
    "      - run: pip install --user -r requirements.txt\n",
    "\n",
    "      # Build the page intermediate HTML files\n",
    "      - run:\n",
    "          name: Build page intermediate HTML files\n",
    "          command: jupyter-book build .\n",
    "\n",
    "      # Persist the specified paths to be used in subsequent jobs\n",
    "      # (see https://circleci.com/docs/2.0/workflows/#using-workspaces-to-share-data-among-jobs)\n",
    "      - persist_to_workspace:\n",
    "          root: .\n",
    "          paths:\n",
    "            - ./_build/\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that, at the end of this job, we've persisted the contents of the `_build/` folder.\n",
    "This allows us to re-use these contents in subsequent jobs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2: Install Jekyll with Anaconda\n",
    "\n",
    "Now that our **page** HTML files have been built, we can use Jekyll to build\n",
    "the HTML for our entire book. This is useful for two purposes:\n",
    "\n",
    "* previewing your Jupyter Book using CircleCI **artifacts**.\n",
    "* publishing the HTML of your book to someplace online\n",
    "\n",
    "In both cases we need to install Jekyll and build the HTML for the book,\n",
    "so let's first define a **CircleCI command** to do this.\n",
    "\n",
    "The following command will copy over the built page HTML from the previous\n",
    "job, install Miniconda which we'll use for environment management, then\n",
    "install Jekyll and the dependencies needed to build your book's HTML using\n",
    "`conda-forge`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "full_width"
    ]
   },
   "source": [
    "```yaml\n",
    "commands:\n",
    "  prepare_jekyll_installation:\n",
    "    steps:\n",
    "      - checkout\n",
    "      - attach_workspace:\n",
    "          # Must be absolute path or relative path from working_directory\n",
    "          at: /tmp/workspace\n",
    "\n",
    "      # Grab the the built intermediate files from the last step\n",
    "      - run:\n",
    "          name: Copy over built site files\n",
    "          command: |\n",
    "            rm -rf _build\n",
    "            cp -r /tmp/workspace/_build .\n",
    "\n",
    "      # Install miniconda to test install\n",
    "      - run:\n",
    "          name: install miniconda\n",
    "          command: |\n",
    "            export MINICONDA=$HOME/miniconda\n",
    "            echo \"export PATH=$MINICONDA/bin:$PATH\" >> $BASH_ENV\n",
    "            source $BASH_ENV\n",
    "            hash -r\n",
    "            wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh\n",
    "            bash miniconda.sh -b -f -p $MINICONDA\n",
    "            conda config --set always_yes yes\n",
    "            conda update conda\n",
    "            conda info -a\n",
    "            conda create -n testenv python=3.7.0\n",
    "            source activate testenv\n",
    "            rm miniconda.sh\n",
    "\n",
    "      # Install Ruby/Jekyll dependencies\n",
    "      - run:\n",
    "          name: Installing Ruby/Jekyll from conda-forge\n",
    "          command: conda install -c conda-forge rb-github-pages\n",
    "\n",
    "      # Build the book's HTML w/ the base_url for CircleCI artifacts\n",
    "      - run:\n",
    "          name: Install book Ruby dependencies\n",
    "          command: bundle install\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "full_width"
    ]
   },
   "source": [
    "## Step 3: Build and preview your book's HTML with Circle artifacts\n",
    "\n",
    "We'll re-use the command from above in order to preview our built site\n",
    "using Circle artifacts. Add the job below to your CircleCI configuration file.\n",
    "\n",
    "```yaml\n",
    "jobs:\n",
    "  doc:\n",
    "    docker:\n",
    "      - image: circleci/python:3.7-stretch\n",
    "    steps:\n",
    "      - prepare_jekyll_installation\n",
    "      - run:\n",
    "          name: Build the website\n",
    "          command: bundle exec jekyll build --baseurl /0/html/\n",
    "\n",
    "      # Tell Circle to store the documentation output in a folder that we can access later\n",
    "      - store_artifacts:\n",
    "          path: _site/\n",
    "          destination: html\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After the Jekyll installation command, it then runs the build command from Jekyll,\n",
    "which outputs all the HTML for your site. We add the `--baseurl /0/html` because this\n",
    "is the prefix for the Jekyll artifact URL. Finally, we use the `store_artifacts`\n",
    "command to tell Jekyll to keep these artifacts for later. Once this job completes,\n",
    "you'll be able to click the \"Artifacts\" tab to preview your book HTML.\n",
    "\n",
    "## Step 4: Automatically publish live HTML from your master branch\n",
    "\n",
    "You can also choose to automatically push built HTML from your **master branch**\n",
    "to a *live* textbook. This lets you automatically deploy changes to your book\n",
    "so that they go live online. Again, we'll use the page HTML command defined above.\n",
    "The job will be very similar to the HTML artifacts preview job, with an extra\n",
    "step to actually **push** the book's HTML online.\n",
    "\n",
    "This step assumes that you are hosting your *live* book on a Git repository\n",
    "using GitHub Pages. We'll need the include a security key that allows CircleCI push access\n",
    "to your GitHub repository. You should first create an SSH key with write access to the\n",
    "repository that will be hosting your live site. Then, use the below configuration\n",
    "to tell Circle to automatically push to this repository."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "full_width",
     "jekyll-raw"
    ]
   },
   "source": [
    "```yaml\n",
    "jobs:\n",
    "  deploy:\n",
    "    docker:\n",
    "      - image: circleci/python:3.7-stretch\n",
    "    steps:\n",
    "      # Add deployment key fingerprint for CircleCI to use for a push\n",
    "      - add_ssh_keys:\n",
    "          fingerprints:\n",
    "            - \"{{ YOUR SSH KEY FINGERPRINT }}\"\n",
    "\n",
    "      - prepare_jekyll_installation\n",
    "      - run:\n",
    "          name: Build the website for deploy\n",
    "          command: bundle exec jekyll build\n",
    "\n",
    "      # Deploy the built site with ghp-import\n",
    "      - run:\n",
    "          name: Deploying site using ghp-import\n",
    "          command: |\n",
    "            pip install ghp-import\n",
    "            ghp-import -p -f -n ./_site/\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "jekyll-raw"
    ]
   },
   "source": [
    "In this case we've used the excellent [ghp-import tool for pushing to github pages](https://github.com/davisp/ghp-import).\n",
    "The command pushes the contents of `./_site` (your book's HTML) to the `gh-pages`\n",
    "branch of the repository. The `-n` flag adds a `.nojekyll` file to the built HTML,\n",
    "which ensures that Jekyll will treat it as raw HTML."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4: Tying these workflows together\n",
    "\n",
    "Now that we've defined several jobs above, we need to tell CircleCI how to\n",
    "use them sequentially (or in parallel). In particular, we want the job\n",
    "that builds each page's HTML to run *first* so\n",
    "that the each page's HTML can be stitched together into a book.\n",
    "Here's the configuration for this:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "full_width"
    ]
   },
   "source": [
    "```yaml\n",
    "workflows:\n",
    "  version: 2\n",
    "  default:\n",
    "    jobs:\n",
    "      - build_page_html:\n",
    "          filters:\n",
    "            branches:\n",
    "              ignore:\n",
    "                - gh-pages\n",
    "      - doc:\n",
    "          requires:\n",
    "            - build_page_html\n",
    "          filters:\n",
    "            branches:\n",
    "              ignore:\n",
    "                - gh-pages\n",
    "      - deploy:\n",
    "          requires:\n",
    "            - build_page_html\n",
    "          filters:\n",
    "              branches:\n",
    "                only:\n",
    "                  - master\n",
    "                ignore:\n",
    "                  - gh-pages\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Appendix: The full configuration file\n",
    "\n",
    "Below is a full CircleCI configuration file that covers each of the steps\n",
    "above. Note that the syntax may be slightly different because we're putting each\n",
    "step above in a *single* file."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "full_width",
     "jekyll-raw"
    ]
   },
   "source": [
    "```yaml\n",
    "# NOTE: This is an example CircleCI configuration that\n",
    "# will build your book and preview its HTML content.\n",
    "# You will probably have to modify it in order to get it working\n",
    "# just the way you want. See https://jupyterbook.org/advanced/circleci.html\n",
    "# for more information\n",
    "version: 2.1\n",
    "jobs:\n",
    "  build_page_html:\n",
    "    docker:\n",
    "      - image: circleci/python:3.7-stretch\n",
    "    steps:\n",
    "      - checkout\n",
    "      - run: pip install --user -r requirements.txt\n",
    "      - run:\n",
    "          name: Build site intermediate files\n",
    "          command: jupyter-book build .\n",
    "\n",
    "      # Persist the built files for the deploy step\n",
    "      - persist_to_workspace:\n",
    "          root: .\n",
    "          paths:\n",
    "            - _build/\n",
    "\n",
    "  doc:\n",
    "    docker:\n",
    "      - image: circleci/python:3.7-stretch\n",
    "    steps:\n",
    "      - prepare_jekyll_installation\n",
    "      - run:\n",
    "          name: Build the website\n",
    "          command: bundle exec jekyll build --baseurl /0/html/\n",
    "\n",
    "      # Tell Circle to store the documentation output in a folder that we can access later\n",
    "      - store_artifacts:\n",
    "          path: _site/\n",
    "          destination: html\n",
    "\n",
    "  # Deploy the built site to jupyter-book.github.io\n",
    "  deploy:\n",
    "    docker:\n",
    "      - image: circleci/python:3.7-stretch\n",
    "    steps:\n",
    "      # Add deployment key fingerprint for CircleCI to use for a push\n",
    "      - add_ssh_keys:\n",
    "          fingerprints:\n",
    "            - \"{{ YOUR SSH FINGERPRINT }}\"\n",
    "\n",
    "      - prepare_jekyll_installation\n",
    "      - run:\n",
    "          name: Build the website for deploy\n",
    "          command: bundle exec jekyll build\n",
    "\n",
    "      # Deploy the built site with ghp-import\n",
    "      - run:\n",
    "          name: Deploying site using ghp-import\n",
    "          command: |\n",
    "            pip install ghp-import\n",
    "            ghp-import -p -f -n ./_site/\n",
    "\n",
    "\n",
    "# Tell CircleCI to use this workflow when it builds the site\n",
    "workflows:\n",
    "  version: 2\n",
    "  default:\n",
    "    jobs:\n",
    "      - build_page_html:\n",
    "          filters:\n",
    "            branches:\n",
    "              ignore:\n",
    "                - gh-pages\n",
    "      - doc:\n",
    "          requires:\n",
    "            - build_page_html\n",
    "          filters:\n",
    "            branches:\n",
    "              ignore:\n",
    "                - gh-pages\n",
    "      - deploy:\n",
    "          requires:\n",
    "            - build_page_html\n",
    "          filters:\n",
    "              branches:\n",
    "                only:\n",
    "                  - master\n",
    "                ignore:\n",
    "                  - gh-pages\n",
    "\n",
    "commands:\n",
    "  prepare_jekyll_installation:\n",
    "    steps:\n",
    "      - checkout\n",
    "      - attach_workspace:\n",
    "          # Must be absolute path or relative path from working_directory\n",
    "          at: /tmp/workspace\n",
    "\n",
    "      # Grab the the built intermediate files from the last step\n",
    "      - run:\n",
    "          name: Copy over built site files\n",
    "          command: |\n",
    "            rm -rf ./_build\n",
    "            cp -r /tmp/workspace/_build .\n",
    "\n",
    "      # Install miniconda to test install\n",
    "      - run:\n",
    "          name: install miniconda\n",
    "          command: |\n",
    "            export MINICONDA=$HOME/miniconda\n",
    "            echo \"export PATH=$MINICONDA/bin:$PATH\" >> $BASH_ENV\n",
    "            source $BASH_ENV\n",
    "            hash -r\n",
    "            wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh\n",
    "            bash miniconda.sh -b -f -p $MINICONDA\n",
    "            conda config --set always_yes yes\n",
    "            conda update conda\n",
    "            conda info -a\n",
    "            conda create -n testenv python=3.7.0\n",
    "            source activate testenv\n",
    "            rm miniconda.sh\n",
    "\n",
    "      # Install Ruby/Jekyll dependencies\n",
    "      - run:\n",
    "          name: Installing Ruby/Jekyll from conda-forge\n",
    "          command: conda install -c conda-forge rb-github-pages\n",
    "\n",
    "      # Build the book's HTML w/ the base_url for CircleCI artifacts\n",
    "      - run:\n",
    "          name: Install book Ruby dependencies\n",
    "          command: bundle install\n",
    "```"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {},
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
